DOCUMENT RESUME 



ED 448 168 



TM 032 129 



AUTHOR 

TITLE 



SPONS AGENCY 
PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Craig, James R.; Kacer, Barbara A. 

Using an Innovation Configuration Component Map To Assess 
the Relationship between Student Achievement and the Degree 
of Implementation of Extended School Services in a Sample of 
Kentucky Middle Schools. 

Kentucky Inst, for Education Research, Frankfort. 

2000 - 11-00 

2 7p . ; Paper presented at the Annual Meeting of the Mid-South 
Educational Research Association (28th, Bowling Green, KY, 
November 15-17, 2000) . 

Reports - Evaluative (142) -- Speeches/Meeting Papers (150) 

MF01/PC02 Plus Postage. 

Academic Achievement; Achievement Gains; * Ancillary School 
Services; *Educational Innovation; Elementary Secondary 
Education; *Maps ; Middle School Students; Middle Schools; 
*Program Evaluation 

* Innovation Configurations; * Kentucky Education Reform Act 
1990 



ABSTRACT 



One of the diagnostic tools used in the evaluation of 
programs implemented under the Kentucky Education Reform Act (KERA) (1990) is 
the Innovation Configuration Component (ICC) Map, designed to specify the key 
elements associated with an innovation and to define what "use" means in the 
context of implementing an innovation. This study investigated whether the 
ICC Map for Extended School Services (ESS) developed by the Kentucky 
Institute for Educational Research could be used to describe the extended 
school services used in middle school instruction in Kentucky, and, if the 
map could be used, the nature of the relationship between the level of 
implementation of extended school services and the state's high stakes 
assessment of a school's students' academic performances as reflected in the 
state's Accountability Index. The sample consisted of 10 middle schools at 5 
levels of achievement. In this sample, there was a positive relationship 
between the implementation of extended school services that provide 
additional instructional or learning opportunities for students and the 
achievement gains posted by the school's students. It is possible that the 
degree of implementation of extended school services in a school is a 
barometer of the school's general willingness to implement a particular KERA 
reform initiative, and not a function of extended school services per se. The 
findings also suggest that another iteration in the formulation of the ICC 
Map of ESS is appropriate. A copy of the revised Map is attached. (Contains 
13 references.) (SLD) 
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Introduction 



The Kentucky Educational Reform Act (KERA) of 1990 is a massive education reform 
bill that specified many new initiatives to be developed and implemented in order to improve the 
system of public education in the state of Kentucky. The effectiveness of those initiatives has 
been evaluated in a variety of ways by different public and private organizations and 
individuals.The Kentucky Institute for Education Research, a private, non-profit research 
organization, has supported a three-year longitudinal study to investigate the education 
innovations being implemented in a small sample of middle and high schools across the state of 
Kentucky. 

The Concerns Based Adoption Model (CBAM), a model of educational innovation and 
change described by Hall and Hord (1987), has been used in this and previous studies (Craig, 
1997; Craig & Kacer, 1999; Craig, Kacer, & Evans, 1998; Craig & Pankratz, 1996) to provide 
the theoretical framework for the examination of the implementation of reform initiatives. One 
of the diagnostic tools used in CBAM to inform and describe the change process is termed an 
Innovation Configuration Component (ICC) Map. A map is designed to specify the key 
components associated with an innovation and to define what “use” means in the context of 
implementing an innovation. Through the Kentucky Institute for Education Research, six 
different ICC Maps have been created (Kentucky Institute for Educational Research, 1996a, 
1996b, 1996c, 1996d, 1996e, 1996f, 1996g)to describe the implementation of education reform 
initiatives in Kentucky regarding: 

■ professional development of the school staff 

■ extended school services 

■ school-based decision making councils 

■ high school restructuring 

■ family resource and youth service centers 

■ educational technology 

■ the primary program 

For example, on the revised ICC Map for Educational Technology (Kacer & Craig, 1999), 
several key innovation components have been identified (e.g., “Teachers use and build upon 
keyboarding, spreadsheets, word processing, and Internet knowledge and skills by incorporating 
their use into regular classroom instruction and learning assignments): 

The descriptive categories associated with this component are: 



(3) 


(b) 


(c) 


(d) 


All(most) teachers do 


Some teachers do 


A few teachers do 


Teachers don’t do it 



Of the four categories associated with the regular use of a keyboarding, spreadsheets, word 
processing, and Internet knowledge in everyday classroom instruction and learning assignments, 
category (a) is considered the ideal implementation description where all teachers (or almost all) 
use keyboarding, spreadsheets, word processing, and Internet knowledge in everyday classroom 
instruction and learning assignments. In contrast, category (d) describes the least desirable 
implementation of keyboarding, spreadsheets, word processing, and Internet knowledge in 
support instruction. The other categories describe intermediary points along the continuum. 
Therefore, if through interviews and observations it is determined that category (a) best describes 
O 
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how the reform initiative of educational technology operates in a school, then that school is 
considered to have completely implemented that particular component of educational technology. 

The maps serve as the basis for an interview (or interviews) with knowledgeable 
administrators and/or teachers and, in some instances, direct observation of activities or review 
of school documents to gain information about the implementation of key features associated 
with particular KERA initiatives. Based on the interview(s) and direct observations and 
document review(if any), the interviewer determines the degree of implementation of 
components associated with an innovation. An innovation profile or score total (which may or 
may not have much meaning) can then be determined for each school regarding each initiative. 

Purpose 

The general intent of the three-year longitudinal study of middle and high schools in 
Kentucky has been to: 

■ Identify key factors that differentiate schools making progress toward 
achievement goals from those that are not 

■ Identify key factors that contribute to successful intervention in low performing 
schools and/or schools in crisis 

The particular focus of the research effort reported here was two-fold. First, the intent 
was to assess whether the ICC Map for Extended School Services(ESS) (Kentucky Institute for 
Educational Research, 1 995) developed by Kentucky Institute for Educational Research could be 
effectively used to describe extended school services employed in instruction in middle schools 
in Kentucky and, second, if the map could be used, the nature of the relationship between the 
level of implementation of extended school services and the state’s high-stakes assessment of a 
school’s students’ academic performances as reflected in the state derived Accountability Index 
(see Instrumentation below). 



Sample 



The sample that was being followed longitudinally consisted of ten middle schools from 
across the state of Kentucky. The schools sampled represent five levels of school performance 
based on the initial pattern of change from 1993 to 1995 of each school’s Kentucky Instructional 
Results Information System (KIRIS) Accountability Index (see below) for students’ academic 
performances. There levels were: 

■ Level 1. (Schools Moving Up) Schools with improving KIRIS scores which had 
baseline KIRIS scores in the upper quartile of all middle or high schools assessed. 

■ Level 2. (Schools Not Moving Up) Schools with no improvement in KIRIS 
scores which had baseline KIRIS scores in the upper quartile of all middle or high 
schools assessed. 

O 
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■ Level 3. (Schools Moving Up) Schools with improving KIRIS scores which had 
baseline KIRIS scores approximately equal to the mean of all middle or high 
schools assessed. 

■ Level 4. (Schools Moving Up) Schools with improving KIRIS scores which had 
baseline KIRIS scores in the lower quartile of all middle or high schools assessed. 

■ Level 5. (Schools Not Moving Up) Schools with no improvement in KIRIS 
scores which had baseline KIRIS scores in the lower quartile of all middle or high 
schools assessed. 

The KIRIS accountability indices for the baseline year and each year since 1993 for the 
ten middle schools in the sample as determined by the state are presented in Table 1. Because of 
a change in the state accountability system, a KIRIS Accountability Index will not be determined 
for 1998. Furthermore, the extent to which the schools as these various levels are representative 
of all middle schools in the state with similar KIRIS Accountability Index values is not known. 
School #3 declined to participate in the third round of data collection. 



Table 1. The KIRIS Accountability Index values and initial 
performance levels for the ten middle schools sampled. 



School 


Level 


Base 


Index 

93 


Index 

94 


Index 

95 


Index 

96 


Index 

97 


Index 

98* 


1 


L3 


40.9 


33.4 


48.2 


56.3 


55.7 


55.8 




2 


L4 


32.8 


29.4 


36.1 


36.2 


30.6 


36.1 




3 


LI 


43.5 


38.8 


48.1 


55.1 


50.9 


54.8 




4 


L4 


34.0 


29.6 


38.5 


45.1 


39.8 


36.6 




5 


L5 


33.3 


33.5 


33.1 


32.7 


30.0 


31.2 




6 


L4 


33.9 


31.3 


36.7 


44.3 


36.6 


39.9 




7 


L3 


38.6 


33.6 


44.1 


51.0 


45.0 


46.6 




8 


LI 


45.0 


42.4 


47.2 


51.5 


46.7 


47.6 




9 


L5 


31.0 


31.9 


29.9 


34.4 


35.5 


32.4 




10 


L2 


45.3 


45.6 


41.8 


41.6 


37.7 


50.5 





*The KIRIS Accountability Indices were not calculated for 1998 by the Kentucky Department of 
Education due to changes in the state’s accountability system. 

School #3 declined to participate in the third round of data collection. 
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Instrumentation 



The ICC Map for Extended School Services 

First, the ICC Map for ESS (Kentucky Institute for Educational Research, 1996b) was 
used to assess after school (and perhaps intersession) instructional services provided to boost the 
academic achievement of students who need additional time/help in mastering academic content. 
An example of an item on the map that is focused on the nature of the instruction delivered in 
ESS is presented in Table 2 below. [The map is available from the Kentucky Institute for 
Education Research at the University of Kentucky in Lexington, Kentucky.] 

Table 2. An item from the ICC Map for Extended School Services 



[diversit 


41 Instructional practices in ESS Droarams. 
focused on needs, collaboration, assessment, and feedback] 


(a) 

A variety of instructional 
strategies are used to 
meet the needs of 
students; (such as 
individual instruction, 
computer assisted 
instruction, peer and 
cross-age tutoring, 
learning skills instruction, 
small group instruction, 
cooperative learning 
groups, active learning). 
There is collaboration 
among. students, parents, 
teachers, and 
administrators. Complete 
student records, including 
referral forms, goals, 
parent permission, 
samples of work, 
assessment information, 
and attendance are used 
to plan instruction.. 


(b) 

Instructional practices 
demonstrate minimal use 
of diverse strategies with 
limited individualized 
instruction, use of 
technology, student 
grouping, and use of 
teacher directed activities. 
There is limited use of 
student records for 
instructional planning. 


(c) 

The ESS program uses 
whole group instruction 
addressing general needs. 
There is no diversity in 
instructional practices. 
Student records are not 
kept for instructional 
practices. 


(d) 

The students complete 
homework assignments 
with minimal guidance. 
The teacher supervises 
the room. 

There is no direct 
assistance. 

Only attendance records 
are maintained. 



fKIRIS^ Accountability Index 

The state of Kentucky has developed the Kentucky Instructional Results Information 
System (KIRIS) to determine a school’s Accountability Index (see below) for students’ academic 
performances. [The system is currently undergoing revision and a new system for determining 
an accountability index for a school will be created.] In the past, a school’s Accountability 
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Index has been created by the state by combining the assessment of the academic achievement of 
students attending a school with several “non-cognitive” factors (e.g., drop-out rate) into one 
measure of a school’s overall performance. The KIRIS Accountability Index values for schools 
are made generally available to the public by the Kentucky Department of Education. 

Data Collection Procedures 

During the third year of the longitudinal study, the ESS coordinators in nine of the middle 
schools in the sample were interviewed on site using the ICC Map for ESS. In most instances, 
facilities were observed as part of the interview. One middle school (a Level 2 school) declined 
to participate in this round of data collection. 

Findings 



Extended School Services 

The ICC Map for Extended School Services is scored from 4 for Category (a) to 1 for 
Category (d) for Item four category items and from 5 for Category (a) to 1 for Category (e) for 
five category items. Therefore, the extended school services implementation scores vary from a 
maximum of possible of 47 to a minimum of 15. The ESS implementation score data for the 
middle schools sampled are presented in Table 3. 

Table 3. School educational technology map 
implementation score and associated comments for the middle schools sampled. 



School 

# 


1999 Extended 
School Services 
Implementation 
Score* 


KIRIS 

Accountability 
Index for 1993 


KIRIS 

Accountability 
Index for 1997 


Accountability 
Index Gain or 
Loss 

(1997-1993) 


Three Year 
Average % 
Free & 
Reduced 
Lunch 


1 


43 


33.40 


55.80 


22.40 


22.00 


2 


38 


29.40 


36.10 


6.70 


59.33 


3 


42 


38.80 


54.80 


16.00 


18.33 


4 


29 


29.60 


36.60 


7.00 


55.67 


5 


29 


33.50 


31.20 


-2.30 


66.33 


6 


38 


31.30 


39.90 


8.60 


63.67 


7 


40 


33.60 


46.60 


13.00 


61.33 


8 


31 


42.40 


47.60 


5.20 


19.33 


9 


39 


31.90 


32.40 


.50 


80.67 



♦The higher the Extended School Services Implementation Score, the closer the implementation of the innovation 
is to what is considered ideal. 
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Student Achievement 



The achievement of the students in a school is reflected in a school’s KIRIS 
Accountability Index as determined by the state. The gain or loss of the nine middle schools’ 
accountability indices is presented in Table 4 below. Some schools posted significant gains over 
the period (e.g.. School #1 gained 22.4 points in its Accountability Index) while others showed 
losses (e.g.. School #5 lost 2.3 points) or very minimal gains (e.g., School #9 gained .5 points). 

Table 4. Gain or loss in a school’s KIRIS 
Accountability Index from 1993 to 1997. 



School # 


KIRIS Accountability 


KIRIS Accountability 


Accountability Index Gain or 




Index for 1 993 


Index for 1997 


Loss (1997-1993) 


1 


33.40 


55.80 


22.40 




2 


29.40 


36.10 


6.70 




3 


38.80 


54.80 


16.00 




4 


29.60 


36.60 


7.00 




5 


33.50 


31.20 


-2.30 




6 


31.30 


39.90 


8.60 




7 


33.60 


46.60 


13.00 




8 


42.40 


47.60 


5.20 




9 


31.90 


32.40 


.50 





*The higher the Extended School Services Implementation Score, the closer the implementation of the 
innovation is to what is considered ideal. 
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Relationship between Accountability Index Gain(Loss) 
and ICC Man for ESS Implementation Score 



The accountability indices and ESS implementation scores for the middle schools 
sampled are presented in Table 5 below. Since the sample size is small, statistics and trends 
which describe the relationship between accountability index gains or losses and the ICC Map for 
ESS Implementation Score may not be particularly meaningful. However, it is interesting to note 
that of the middle schools sampled, those with the lowest three year average percentage of 
students on free and reduced lunch showed some tendency to have higher ESS Map scores (rho= 
-.269, p>.05) and larger gains in the KIRIS Accountability Index (rho= -.567, /?>.05). Of greater 
interest is the relationship between ESS Map scores and gains in the KIRIS Accountability Index 
from 1993 to 1997. In Figure 1 below, the ESS Map scores and the KIRIS Accountability Index 
gains(losses) mirror each other fairly well (rho= .740,/><.05). If three year average percentage of 
students on free and reduced lunch is partialed out, the relationship between the ESS Map scores 
and the KIRIS Accountability Index gains(losses) does not appear to change (r=.771, p< 05). 
These data would suggest that for the middle schools sample, the extended school services 
programs that most closely approximate the ideal implementation of that innovation tend to 
produce bigger gains over time in student achievement. 

Table 5. School extended school services map 
implementation score and associated comments for the middle schools sampled. 



School # 


1 999 Extended School 
Services Implementation 
Score* 


Accountability Index 
Gain or Loss ( 1 997- 
1993) 


Three Year Average % 
Free & Reduced Lunch 


1 


43 


22.40 


22.00 


2 


38 


6.70 


59.33 


3 


42 


16.00 


18.33 


4 


29 


7.00 


55.67 


5 


29 


-2.30 


66.33 


6 


38 


8.60 


63.67 


7 


40 


13.00 


61.33 


8 


31 


5.20 


19.33 


9 


39 


.50 


80.67 



*The higher the Extended School Services Implementation Score, the closer the implementation of the 
innovation is to what is considered ideal. 
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1999 ESS Map Score versus 
Accountability Index Difference Score 




m 

Total ESS Map Value 

for 1999 

1 

Acountability Index 
Difference ('97-'93) 
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Discussion 



At least in the small sample of middle schools studied, there is positive relationship 
between the degree of implementation of extended school services that provide additional 
instructional/leaming opportunities for students and the achievement gains posted by the school’s 
students. However, it is not known if the gains are directly a function of the implementation of 
extended school services in a school or simply associated with the attitudes and teaching 
behaviors of teachers in middle schools which have a high degree of implementation of extended 
school services. It is possible that the degree of implementation of extended school services in a 
school is a barometer of a school’s general willingness to implement the a particular KERA 
reform initiative and not a function of extended school services per se. 

The findings also suggest that another iteration in the formulation of the ICC Map of ESS 
is appropriate. Therefore, the researchers have produced a revised ICC Map of ESS, a copy of 
which is attached. 
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School Level Program Design [circle the statement that most accurately applies] 
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Organizing and grouping students [linkage to learning style, learning objective, developmental level] 
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Program evaluation [data based, continuous] 
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Linkages with other KERA strands and other supporting programs [communication, collaboration] 
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